1 research outputs found
Large-scale unsupervised audio pre-training for video-to-speech synthesis
Video-to-speech synthesis is the task of reconstructing the speech signal
from a silent video of a speaker. Most established approaches to date involve a
two-step process, whereby an intermediate representation from the video, such
as a spectrogram, is extracted first and then passed to a vocoder to produce
the raw audio. Some recent work has focused on end-to-end synthesis, whereby
the generation of raw audio and any intermediate representations is performed
jointly. All such approaches involve training on data from almost exclusively
audio-visual datasets, i.e. every audio sample has a corresponding video
sample. This precludes the use of abundant audio-only datasets which may not
have a corresponding visual modality (e.g. audiobooks, radio podcasts, speech
recognition datasets etc.), as well as audio-only architectures that have been
developed by the audio machine learning community over the years. In this paper
we propose to train encoder-decoder models on more than 3,500 hours of audio
data at 24kHz, and then use the pre-trained decoders to initialize the audio
decoders for the video-to-speech synthesis task. The pre-training step uses
audio samples only and does not require labels or corresponding samples from
other modalities (visual, text). We demonstrate that this pre-training step
improves the reconstructed speech and that it is an unexplored way to improve
the quality of the generator in a cross-modal task while only requiring samples
from one of the modalities. We conduct experiments using both raw audio and mel
spectrograms as target outputs and benchmark our models with existing work.Comment: Submitted to IEE